Temporal Psychovisual Modulation: a new paradigm of information display

We report on a new paradigm of information display that greatly extends the utility and versatility of current optoelectronic displays. The main innovation is to let a display of high refresh rate optically broadcast so-called atom frames, which are designed through non-negative matrix factorization to form bases for a class of images, and different viewers perceive selfintended images by using display-synchronized viewing devices and their own human visual systems to fuse appropriately weighted atom frames. This work is essentially a scheme of temporal psychovisual modulation in visible spectrum, using an optoelectronic modulator coupled with a biological demodulator. Information display technologies have played an indispensible role in the advancement of human civilization, from rock carving of prehistoric tribesmen to pervasive use of electronic displays in modern societies. The ever-presence, great variety and continuously increasing sophistication of electronic displays profoundly shape the ways we learn, work, communicate, entertain, travel, etc. and even how we socially behave. This article presents a new paradigm of information display that is conceived by making an ingenious interplay of critical flicker frequency of human vision, high refresh-rate digital display and optoelectronic viewing devices. The new display paradigm, called temporal psychoviusal modulation (TPVM), differs fundamentally, in design principle, user experience and cost effectiveness, from the head-mounted display (HMD) technology (1,2), while offering the same functionalities and more. TPVM exploits a well known phenomenon in psychophysics of vision: the human visual system (HSV) cannot resolve temporally rapidly changing optical signals beyond flicker fusion frequency (around 60 Hz for most viewers and under most conditions (3)). For the purposes of this research, the good news is that modern digital displays can run at much higher refresh rates, e.g. 120 Hz, 240Hz and beyond. For example, the new light modulators such as the deformable mirror devices and grating light valve devices can lead to very high refresh rate (up to 88k Hz) and spatial resolution for digital projectors; the main stream LCD/ LED technology also achieves 120Hz or 240Hz refresh rate, as demanded by emerging applications of 3D video playback. If the refresh rate of a digital display exceeds the limit of temporal resolution of HSV, then it is possible for HVS to “render” or perceive different images by fusing differently weighed consecutive frames (called atom frames) emitted from the same display. These psychovisual signal processing operations and the resulting net visual effects can be mathematically modeled as follows. Let d f be the refresh rate of a digital display and v f the critical flicker frequency at which HVS just fails to discern discrete frames. For simplicity, assume d v f Mf  , where M is a positive integer. Suppose that 1 2 , , , K y y y are the K images to be formed by HSV. These K target images constitute an N K  matrix Y with column k being image K y of N pixels. Temporal psychovisual modulation is a problem of signal decomposition  Y XW where X is an N M  matrix whose columns are M atom frames 1 2 , , , M x x x of N pixels to be cyclically displayed at frequency d f , and W is an M K  modulation weighting matrix. In other words, image ,1 k k K   y , is psychovisually formed as a linear combination of the M atom frames. In the above problem formulation and in the sequel an assumption is made that the display is psychovisually linearized.